Case study: selecting a Tier 3 data center for server migration in Europe

We first met the client we are going to talk about today at a trade fair that was held in 2015 and focused on data center services and facilities. The representatives of the companies met, and it turned out that the acquaintance was very timely for our future client. It was at this point that he was looking for the option of moving servers to Germany, Frankfurt to be exact. Why here, we will tell you about it below. First, let’s have a look at the story of the customer and the reasons for migration.

We have no right to disclose the names of our clients, so we will call the company iGaming for convenience. Their business niche is gambling and cyber sports. The company has an online gambling platform, and it also develops gaming platforms and applications. So, this name captures the essence of their activity.

From a small company to global business for 8 years

When we started our cooperation, the company was a small business located in one of the European countries. In a few years, it has turned into a huge gaming empire with a geo-distributed infrastructure, powerful modern equipment, and high fault tolerance. Their business has scaled internationally and today provides online gaming service to many users around the world. We have witnessed their impressive growth dynamics and are proud to say that we have been helping to build and develop their IT system for 8 years.

Prerequisites for moving to a data center in another country

At the time we met, the main node of iGaming infrastructure was in Poland. They deliberately moved the node to a neighbouring country because of constant troubles at home: inspectors checking the companies that rented racks in the same data center (DC) as our future customer could confiscate other company’s equipment, including iGaming servers. After clarifying the circumstances, the equipment not related to violators was returned to its owners. But no one covered the associated downtime. Regular confiscations created high risks for the customer, which were certainly a no go for them: a gambling platform providing online services can’t stand idle.

If servers become unavailable even for an hour, the business activities are completely stopped, which means losing customers and making no profit.

Due to high risks, it was decided to locate the equipment for the company’s main services in one of the data centers in Poland. There were several reasons to choose this country:

• firstly, law enforcement agencies in Europe are not allowed to seize any equipment — computers or servers — from companies they check,
• secondly, this option had geographical and commercial advantages.

Therefore, the core of the company’s services was transferred to the Polish data center. Physical servers in Poland were used to store their main infrastructure data to make it available at any time.

Renting servers was inexpensive, and a remote location in a neighbouring European country with different legislation helped to avoid problems with illegal equipment seizure.

How the migration of servers to Poland ended

The client was looking for security and stability, and the migration of servers was supposed to play a role of insurance. However, the move to Poland, as it turned out, was not the best way out.

The availability of servers for a gaming platform is vital. Lack of stable availability has a negative impact on business.

Sometimes, their site suddenly became unavailable when their users made payments (made a bet, opened a deposit, transferred money to their deposit). This meant a fall in the site rating, more distrust to the company and customers going to competitors. Their turnovers reduced, which entailed financial losses and reputation damages. The company lost a lot of money. When a business relies on 100% availability of resources, any minor mistakes can lead to serious consequences. This is how the head of the IT department of iGaming explained the reasons why they decided to move the servers with the security kernel to another location.

It cannot be said that all data centers in Poland are bad. It may be that the SLA of the selected DC was not the right one or that the availability issues were due to accidents. Nevertheless, the customer was not lucky with these negative cases both in the homeland and in the neighbouring country. With this in mind, they have developed certain requirements for a new data center. First of all, the level of the data center reliability standard was very important (only Tier III), high service level, certification. That is, the customer was looking for a data center that can provide maximum reliability, fault tolerance and performance to safely host business-critical services.

How it all started

Just as iGaming faced the need to choose a new data center for server migration, they accidentally met our company representatives at the exhibition. They shared their difficulties in a conversation and our development director described how we can help.

The customer was interested in a cloud solution. iGaming developed a lot of software products for cloud solutions suitable for Oracle Cloud, they used the Oracle database and had whole physical modules with Linux Oracle (which they supported for a long time until they completely switched to virtual machines). And we had Oracle virtualisation. This meant good compatibility.

At that time, it wasn’t about cooperation because the customer was choosing the most suitable data center to rent cloud services for their gaming platform and we were one of the potential partners. But they got interested in possible prospects and decided to continue communication.

‘Live’ negotiations

At first, we communicated remotely by phone and e-mail. It is usually our project coordinator who carries out the most of communication with existing and potential customers, but a face-to-face of technical specialists is necessary for better understanding. We decided to organise negotiations in the customer’s office, so our project coordinator together with the technical director came to iGaming.

We agreed to conduct business negotiations in the customer’s office because it was important to:

• first, get to know each other better,
• secondly, meet with the management, the specialists responsible for infrastructure and networking in person.

When it is a live conversation, technical specialists can use their usual workplace language and discuss all the nuances impossible to consider when communicating online. That is why we have introduced a kind of internal SLA: when a customer contacts us, we get acquainted with them or come to them in one or two weeks.

Olga Boujanova

iGaming was no exception. It was very important for us to introduce our technical director, an ideologist and architect of our infrastructure, a major virtualisation specialist. The company had quite a lot of personnel at that time, and it was easier to organise negotiations in their office. The meeting appeared to be rather productive. More than 10 customer’s people from all departments took part in it: technical specialists, developers, infrastructure administrators, network engineers and, of course, security specialists.

Negotiations lasted more than 3 hours, we covered a lot of technical issues and came to understand that both sides speak the same language, have same-level professionals and grounds for mutually beneficial cooperation.

Contract signing

The work of the node in Poland was unstable, which was a critical issue, so the customer was in a hurry. Immediately after returning from the negotiations, we began to work on the project preparation. We introduced all the important terms and requests of the company into the contract, specifying the service level we were obliged to provide, the customisation and server configuration requirements.

The contract was approved and signed remotely. They have their own system for confirming approvals, it was enough to send a scanned copy of the document. It suited us too.

All the preliminary work required some time. It took two months from the meeting in the customer’s office until the signing of the final version of the contract and the allocation of resources.

What helped our business partnerships

Face-to-face communication with the company, customised solutions for their problems, convenient and safe location in Europe — all these led to the signing of a long-term contract and successful 8-year interaction.

It is important to mention that our face-to-face meeting and personal acquaintance with the head of IT infrastructure were the key factors for the customer in choosing us. But there were other reasons too.

• The location of our resources in the Telehouse Frankfurt data center.
• Our latency with good ping speed (30 milliseconds to the customer’s country).
• The data center certifications are suitable for obtaining gambling business licences.

Now more details about the main points.

Why select a certified Tier III data center in Frankfurt

The customer was looking for the safest and most reliable place for servers in terms of availability and considered our city for moving. And here is why.

The activities of financial and military institutions directly depend on the availability of reliable information and computer networks and communications. Frankfurt is the best place in Europe in these terms.

Historically, Frankfurt has been the center of Germany’s financial life. The city is known as the largest telecommunications center of the country and the European banking capital with the following located here:

• European Central Bank and many other financial institutions,
• DE-CIX — the largest commercial traffic exchange node (500+ Internet providers and telecom operators),
• more than 60 data centers with their number constantly increasing.

In addition, US military bases had been located in Frankfurt for a long time.

That is why Frankfurt has modern fiber optic communication lines and developed communications — to make all of these run smoothly (not to mention backup communications of the military for additional security guarantees). Not surprisingly, many companies are trying to locate their IT resources — physical and virtual cloud servers — in Frankfurt for better availability.

Preliminary testing of our services

In order for potential customers to make sure that the data center performance meets their business goals, we provide a free speed test and give access to virtual machines to check latency. It is our common practice, and iGaming specialists tested all the important characteristics and made sure that we were a good fit for them before signing the contract.

One of the exchange points of our providers is located in DE-CIX and all of them are distributed: some providers cover Eastern, Central Europe and the UK, others cover the United States. We have carefully chosen our uplinks to provide the best latency throughout the map and the best coverage from both sides.

Main characteristics of the data center

The projects we work with are hosted in Telehouse Frankfurt. This is a highly reliable European data center, one of the data centers of Telehouse with its headquarters in London. Telehouse is part of KDDI Corporation, the largest Japanese corporation providing telecommunications services around the world.

Telehouse Frankfurt is a complex of buildings (25 thousand square metres in total) and is connected to DE-CIX. The level of reliability of Telehouse Frankfurt infrastructure and equipment is Tier III, which is optimal for cloud services.

Data centers that comply with this standard have extensive backup capacities and provide for fully-redundant cooling systems. Their reliability is very high. To create reliable fault-tolerant infrastructure in Tier III data centers, there is no need to install additional equipment, these DCs can work smoothly. The system availability is 99.999%.

PCI, ISO, green energy, and other data center standards

The customer’s use of resources in our certified Tier III data center allows them to meet ISO 9000 certification requirements and obtain licences necessary for their activities.

Telehouse data center meets the following standards:

• ISO/IEC 27001:2013,
• ISO 9001:2015,
• DIN EN ISO 50001:2018,
• DIN EN 50600,
• PCI-DSS,
• ISAE 3402 Type 2,
• Green Energy.

First step: technical adjustment

After discussing all the details and signing the contract, we started implementing the project. First, the customer took our cloud solution built with Oracle virtualisation. We provided a pool of 5 servers and a separate data storage system connected to them via Fibre Channel. We also provided them with access to the control panel. iGaming engineers performed the configuration and setup of virtual servers on their own.

It took some time to adjust with each other: we tuned up the processes, solved various current issues, looked for optimal solutions. Even about 8 years ago, the customer’s team consisted of competent and technically savvy IT specialists with relevant education and additional certification in technologies related to the construction of reliable fault-tolerant systems. Many of them had extensive experience in building IT infrastructure in companies with strict requirements for fault tolerance and security.

In most cases, they had an idea of how to build and administer the infrastructure, but sometimes they also needed technical support from us. In particular, they turned to us for advice when their Oracle Database was connected to our Oracle cloud and there were a number of complex technical issues that required a deeper study by field professionals. And we, in turn, had to analyse and evaluate the effectiveness of the solutions used by the customer, as well as consult with developers a lot.

This was how they got to know our processes better and checked our technical expertise. The customer built their infrastructure very carefully in terms of fault tolerance and reliability. With their high-level specialists and the negative experience, they had in other data centers, iGaming perfectly knew the worth of security and were meticulous in details.

Clear communication is always very important for customers, and our project manager paid a lot of attention to this aspect. However, the technical part was even more important. When the customer realised that we understand each other perfectly, are of one mind, our specialists are quick to grasp everything, ready to follow the company’s interests and offer different options to improve the system reliability, our cooperation reached a completely new stage.

We encountered professionals who had a clear idea of what the IT infrastructure should be like. They had our understanding, support and implementation of this idea.

How the customer’s IT infrastructure developed

Since 2015, the customer has been using a personal infrastructure that was built with the help of our cloud solution. At first, they did not order a separate rack, so the equipment was distributed between different racks. Nevertheless, we tried to put most of it in one cage and in one or two communication cabinets to make it more convenient to maintain the equipment.

Their business grew and developed. After about a year and a half or two of cooperation, the company began to expand massively. New challenges and new infrastructure requirements emerged. They needed full resource management.

To find a solution to this problem, we held joint negotiations and decided to use a hybrid infrastructure. In addition to virtual servers, we leased dedicated servers of suitable configurations Then they began to regularly add physical servers. They always had them in multiples of five — 5 or 10 pieces, the largest pool was 25 servers.

With one of the following expansions, separate network equipment and physical storage were included in their IT infrastructure: first, Fujitsu Eternus DX Storage, then Dell Unity XT All-Flash Storage (modern SSD-based stores with increased fault tolerance). The stability of the connections between the equipment was ensured via fiber optic cables via Fiber Channel.

The number of hosts for different types of virtualisation grew rapidly with each coming year. Over time, the customer also began to use the solutions of large cloud providers. To support these solutions, we connected their existing infrastructure in our data center with such large cloud platforms as Microsoft Azure, AWS and Google Cloud.

How new equipment was connected

The equipment was combined and the devices had to be interconnected to create a hybrid infrastructure. We performed the connection and the customer gave their instructions.

The connection of servers was the same as usual: one connection for management, two connections via Gigabit Ethernet ports and a connection via Fibre Channel to the data storage system. In addition, the entire infrastructure was built with two switches to ensure the system’s stability and redundancy. When one switch fails, the system continues its work through the second switch.

We discussed, agreed, and formalised the main connection scheme initially when they first added 5 servers. In the future, when the intensive increase in the capacity of the IT infrastructure began and it became quite large, we already knew the principle. They typically ordered new servers and then routinely discussed the configuration, RAM, CPUs they needed and other small details. Components were added and connected in the established way.

Communication with the customer

Our corporate rules include meetings with existing customers twice a year. We discuss the details of cooperation, the implementation of solutions, plans for the next period. Unfortunately, the pandemic and the changes in the global situation got in the way of this tradition. Therefore, the last face-to-face meeting with iGaming representatives was held in 2019. Now there is no possibility to come to the country where the company’s office is located, and all communications are remote. This makes it difficult to work, however, we still try to be on the same page with the customer. In January of this year, we met online to discuss the improvement of infrastructure, changed requirements, development prospects.

Why January? Firstly, this is connected to the specific character of the company’s gambling activities: the first month is the quiet period for their business. Secondly, the budget was already allocated, the tasks for the near future were set by the owner and the employees responsible for the infrastructure had a clear understanding of what to do.

The purpose of our annual meetings with customers is to discuss plans and implement the goals set for the year, both towards expansion and optimisation, reducing the number of servers, updating the line, installing new processors, etc.

Personalised solutions

When it comes to new products and offers, we always work individually with each customer and their specific situation. We have to take into account their infrastructure. We have no bulk mailing. There are no one-size-fits-all offers, each IT system is unique, and each solution is always personalised.

When we have a new product or options, we check if any of the existing customers may have a need for them. We check the servers involved in the infrastructure. If the innovation we have can potentially be interesting and technically applicable, then we inform the customer. For example, if their physical servers have been used for 5+ years and the platform has become outdated and we have a more powerful solution, we will certainly offer them an optimisation option, new opportunities.

These modifications often become very profitable for customers both in terms of reliability and financial costs. With less modern machines they can reach much higher performance compared to that with outdated equipment.

If we believe that our offer is relevant, we write a letter to the customer, then call and discuss everything in detail. It is because our conversations often reveal important aspects that completely change everything and require other options.

At the same time, we do not insist on upgrades if a customer refuses to do so. We will continue to support their equipment as usual.

Backup storage solution

One of these new solutions we offered to iGaming to improve their infrastructure and optimise resources was Object Storage S3 for storing all their backups.

Typically, customers consider this option to distribute their backups and infrastructure between different data centers. However, in this case, the business began to grow, and the customer had to turn to major cloud providers mentioned above. Object solutions and Object Storage S3 are a good choice in these circumstances.

It made sense to offer them to use this object solution and store backups with us thereby ensuring the required diversity.

How we fought DDoS attacks

In addition to choosing a reliable data center, there is another big problem that affects almost all gambling platforms from time to time — DDoS attacks. The initiators of such attacks, as a rule, are competitors, and this type of cybercrime is well known to everyone who works in this field. Our customer was no exception. Their online platforms were regularly experiencing DDoS attacks.

A DDoS attack is a hacker attack that floods a server with fake traffic and overwhelms it thereby causing a system failure. A dedicated server can withhold user traffic under normal load but crushes under the influence of hackers and the related online service becomes unavailable. With such failures, business owners lose both profits and reputation.

Since online gaming business directly depends on the availability of their online sites, each successful DDoS attack causes huge losses. When hackers attack one customer, they flood the entire communication channel with traffic leading to an overload of the network and upstream routers. So they needed a solution to address these threats.

Because of the rapid growth and development of the customer’s company, they faced DDoS attacks at least twice a year — at peak events. World Championships, European Championships, Grand Prix, Formula 1, Olympic Games, matches, seasonal skiing and figure skating competitions, New Year and Christmas holidays, etc. — all major sports events and holidays with peak user traffic were necessarily accompanied by hacker attacks. The attacks did not last long, not more than a few hours, but the business suffered huge losses.

It was a long way: we explored methods to disrupt attacks, offered different means of DDoS protection. We expanded our infrastructure to withstand this traffic flow and protect the rest of the customers because they suffered from these attacks too. Unfortunately, we had no comprehensive tested solutions suitable for a gambling platform at that time. Our data center did its job well, but the initial configuration of the customer’s infrastructure had vulnerabilities and narrow necks and was not ready to repel cyber-attacks. They asked us to install filters and installed theirs, tried to cope with their own technical means, but this was not enough. We needed a profound solution. Since the company had just begun to grow, they had no allocated funds for this.

We should clarify that DDoS attacks are always funded by someone and can’t last for long. Site owners understand this and make allowance for infrastructure unavailability risks and try to build good DDoS protection. However, such protection is not always the answer. The costs of upgrading the infrastructure to prevent DDoS attacks should be commensurate with the possible losses resulting from attacks. This means that it doesn’t always make sense to pay for this protection. It is necessary to weigh the pros and cons — the losses can be significantly less compared to the cost of protection, so there is no point in spending more if the company is small.

Apparently, our customer did not consider these costs justified at the initial stages of their business development. However, with multiple locations established around the world, they needed extensive DDoS protection for the entire infrastructure. We could offer only our local German solutions, but they did not quite fit as they covered only one node in Frankfurt.

Therefore, as soon as they had the necessary funds, the customer chose comprehensive Cloudflare protection as the most effective and convenient option for online gambling. Many gambling platforms use the services of this provider because it offers not only protection but also good availability in most regions. It has caching tools, various geo-distribution filters, Firewall, licences in different countries of the world. Cloudflare is convenient for gaming platforms in terms of improving the level of service.

For the convenience of the customer and in addition to dedicated servers, now we are constantly providing them with BYOIP, i.e., a service that allows them to use their own IP addresses in the subnet of the cloud provider, as well as lease IPv4 which go through Cloudflare solutions used to protect servers from DDoS attacks.

Olga Boujanova

Profits for the customer: impressive dynamics

The company started its way in 2015 with only two countries, one rack and one licence and it has demonstrated impressive development dynamics during our cooperation. It has grown strongly by becoming a very large and solid company with large turnovers. Their presence has expanded, and the audience has also multiplied. We are no longer the only data center where the critical core of the company’s IT infrastructure is located — now they have nodes all around the world.

Within almost 8 years of our partnership, the customer has acquired opportunities:

• To increase performance and completely modify their IT infrastructure,
• To cover different regions of the world,
• To switch to expensive solutions, such as Cloudflare,
• To significantly expand and enhance their team by inviting new specialists particularly AWS experts.

All of this mean tremendous growth of the business. But the company does not stop there, it is growing and developing further. They need more and more capacities. The customer has big plans for the current year, and we are here to help.

We are constantly working together and figuring out any difficulties they face. We follow market changes, the latest trends, and innovations. We select profitable, convenient, and relevant solutions in a tailored way and offer various flexible options for improving their infrastructure, adapting to the customer’s needs and market conditions.

Mutually beneficial cooperation

The iGaming project helps us to develop as well. To meet the requirements and goals of the business, we are constantly looking for new optimisation methods and fresh solutions, we update our resources. Thanks to this project, our company has entered into a partnership agreement with Cloudflare.

When Cloudflare learned that the company uses our IPv4, their representative contacted our project manager and offered an interesting option for cooperation and customer exchange: we provide IP addresses for rent and they provide network protection against DDoS attacks.

If you are interested in building a reliable fault-tolerant infrastructure for your project, please contact us. We will be happy to consult, answer all questions and create your dream IT infrastructure together.

SET UP A CALL

Article author

Olga Boujanova

Consultant on server hardware and data center organization

Case study:How to move to a new data center with minimal costs by leasing equipment?

READ MORE